Eyeglasses-type wearable device and method using the same

ABSTRACT

An eyeglasses-type wearable device of an embodiment can handle various data inputs. The device includes right and left eye frames corresponding to positions of right and left eyes and nose pads corresponding to a position of a nose. Eye motion detection electrodes (sightline detection sensor electrodes) are provided with the nose pads to detect the eye motion of a user. Transmitter/receiver electrodes (capacitance sensor electrodes) of a gesture detector are provided with a part of the right and left eye frames to detect a gesture of the user. Various data inputs are achieved by a combination of input A corresponding to a gesture of the user detected by the gesture detector and input B corresponding to the eye motion of the user detected by the eye motion detector.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/100,688, filed Nov. 20, 2020, which is a continuation of U.S.application Ser. No. 16/181,255 filed Nov. 5, 2018, now U.S. Pat. No.10,877,567, issued Dec. 29, 2020, which is a divisional of U.S.application Ser. No. 15/821,511, filed Nov. 22, 2017, now U.S. Pat. No.10,168,793, issued Jan. 1, 2019, which is a continuation of U.S.application Ser. No. 14/979,183, filed Dec. 22, 2015, now U.S. Pat. No.9,880,633, issued Jan. 30, 2018, which is based upon and claims thebenefit of priority from Japanese Patent Application No. 2015-172153,filed Sep. 1, 2015, the entire contents of which are incorporated hereinby reference.

FIELD

Embodiments described herein relate generally to an eyeglasses-typewearable device.

BACKGROUND

Some of eyeglasses-type wearable devices (eyewear) detect an eyepotential. In such eyewear, a change in an eye potential caused by eyemotion or eye movements of a user is detected by detection electrodesprovided with nose pads and a bridge (a part in front of the brow of theuser) between eye frames of the glasses. The eye potential changesdepending on types of the eye movements of the user (up-and-down andright-and-left movements and blinks). Using this mechanism, the userwith the eyewear can perform data input corresponding to the types ofthe eye motion or eye movements.

According to the prior art eyewear, the electrode contacting the user isprovided with the bridge, which does not contact a user in ordinaryglasses. That is, in such eyewear, contact points with the face of theuser are not only the nose pads and some user may possibly feeluncomfortable in wearing.

Furthermore, the data input is only made by the eye movements in theprior art eyewear and eyestrain should be considered. Thus, data amount(or the number of data items) which can be input in series and types ofthe data to be input are limited.

Therefore, as a target of the present application, embodiments presentan eyeglasses-type wearable device which can handle various data inputs.

According to an embodiment, the eyeglasses-type wearable device hasright and left eye frames arranged near the positions of right and lefteyes and nose pads arranged at the position of a nose, and the deviceincludes a display provided with at least one of the right and left eyeframes, a gesture detector which detects a gesture indicative of amovement of a user, and an eye motion detector which detects eye motionor eye movement of the user. (Since the display is provided with atleast one of the right and left eye frames, the gesture detector can beprovided with at least one of the right and left eye frames.) The eyemotion detector can be provided with the nose pads, and an electrodecontacting the brow of the user may not be required.

Data input from the gesture detector (data input A) and data input fromthe eye motion detector (data input B) are obtained and a combinationthereof can be used. Input data types can be increased by using such acombination, the eyeglasses-type wearable device which can acceptsvarious data inputs can be achieved.

Furthermore, the data input operation is performed by not only eyemotion or eye movements but also gestures, and thus, eye strain of theuser can be reduced.

BRIEF DESCRIPTION OF THE DRAWINGS

A general architecture that implements the various features of theembodiments will now be described with reference to the drawings. Thedrawings and the associated descriptions are provided to illustrate theembodiments and not to limit the scope of the invention.

FIG. 1 shows an eyeglasses-type wearable device of an embodiment, andshows an example of an arrangement of gesture detection capacitancesensor electrodes (140 to 144).

FIG. 2 shows how to obtain detection voltage signals (Vrxbuf) from achange in a capacitance (Ch) corresponding to gestures.

FIG. 3 shows an eyeglasses-type wearable device of another embodiment,and shows an example of the arrangement of capacitance sensor electrodes(140 to 144 and 141* to 144*) for the gesture detection and an exampleof the arrangement of eye motion detection electrodes (151 a, 151 b, 152a, and 152 b) provided with a nose pad.

FIG. 4 shows an eyeglasses-type wearable device of a still anotherembodiment, and shows another example of the arrangement of capacitancesensor electrodes (140 to 144 and 141* to 144*) for the gesturedetection.

FIG. 5 shows various examples of eye motion detection electrodes (151 a,151 b, 152 a, and 152 b) provided with nose pads.

FIG. 6 shows an example of how to extract detection signals from the eyemotion detection electrodes (151 a, 151 b, 152 a, and 152 b) providedwith nose pads.

FIG. 7 shows a data processor 11 (integrated circuit including, forexample, a processor 11 a, nonvolatile memory 11 b, main memory 11 c,communication processor 11 d, and sensor lie) attachable to theeyeglasses-type wearable devices of various embodiments and peripheraldevices (such as a display 12, camera 13, gesture detector 14, eyemotion detector 15, and power source BAT).

FIG. 8 shows an electro-oculogram (EOG) with respect to a relationshipbetween eye motion from the front to the above and detection signallevels (Ch0, Ch1, Ch2, and average level Ch1+2 of Ch1 and Ch2) obtainedfrom three analog/digital converters (ADCs) of FIG. 6.

FIG. 9 shows an electro-oculogram (EOG) with respect to a relationshipbetween eye motion from the front to the below and detection signallevels (Ch0, Ch1, Ch2, and average level Ch1+2 of Ch1 and Ch2) obtainedfrom three ADCs of FIG. 6.

FIG. 10 shows an electro-oculogram (EOG) with respect to a relationshipbetween eye motion from the left to the right and detection signallevels (Ch0, Ch1, Ch2, and average level Ch1+2 of Ch1 and Ch2) obtainedfrom three ADCs of FIG. 6.

FIG. 11 shows an electro-oculogram (EOG) with respect to a relationshipbetween eye motion repeating blinks (both eyes) for five times with fivesecond intervals and detection signal levels (Ch0, Ch1, and Ch2)obtained from three ADCs of FIG. 6, where the sight is in front.

FIG. 12 shows an electro-oculogram (EOG) with respect to a relationshipbetween eye motion repeating blinks (both eyes) including an eye closingfor one second and an eye opening for four seconds for five times anddetection signal levels (Ch0, Ch1, and Ch2) obtained from three ADCs ofFIG. 6, where the sight is in front.

FIG. 13 shows an electro-oculogram (EOG) with respect to a relationshipbetween eye motion repeating blinks (both eyes) for five times andrepeating left eye winks (blinks of left eye) for five times with eyesfront and detection signal levels (Ch0, Ch1, and Ch2) obtained fromthree ADCs of FIG. 6, where the sight is in front.

FIG. 14 shows an electro-oculogram (EOG) with respect to a relationshipbetween eye motion repeating blinks (both eyes) for five times andrepeating right eye winks (blinks of right eye) for five times with eyesfront and detection signal levels (Ch0, Ch1, and Ch2) obtained fromthree ADCs of FIG. 6, where the sight is in front.

FIG. 15 is a flowchart which shows processes performed by combinationsof data inputs by gestures (data input A) and data inputs by eye motionor eye movements (data input B).

DETAILED DESCRIPTION

Hereinafter, various embodiments will be explained with reference toaccompanying drawings.

These embodiments may relate to various wearable devices including anyof an eyeglasses-type wearable device, a glasses-type wearable device, aspectacle-type wearable device, and the like. In this specification(including detailed description and claims) these various wearabledevices are simply represented by the term “eyeglasses-type wearabledevice” unless otherwise noted. In other words, the term“eyeglasses-type wearable device” should be broadly interpreted as awearable device regarding an eye or eyes.

The “user” used in this specification may have the meaning of “operator”or “worker” in a warehouse.

FIG. 1 shows an exterior of an eyeglass-type wearable device 100 of anembodiment. In this example, a right eye frame (right rim) 101 and aleft eye frame (left rim) 102 are connected by a bridge 103. The rightand left eye frames 102 and 101 and the bridge 103 can be formed of aconductive material such as a lightweight metal (e.g., aluminum alloy ortitanium). The outer left side of the left eye frame 102 is connected toa left temple bar 106 via a left hinge 104 and a left end cover (leftear pad) 108 is provided with the tip of the left temple bar 106.Similarly, the outer right side of the right eye frame 101 is connectedto a right temple bar 107 via a right hinge 105 and a right end cover(right ear pad) 109 is provided with the tip of the right temple bar107.

A data processor 11 (an integrated circuit of a few millimeter square)is embedded in a part of the eye frame 101 near the right hinge 105 (orinside the right temple bar 107). The data processor 11 is an LSI inwhich a microcomputer, memory, communication processor, and the like areintegrated (the data processor 11 will be detailed later with referenceto FIG. 7).

Although this is not depicted in FIG. 1, a small battery such aslithium-ion battery (corresponding to BAT in FIG. 3) is embedded in, forexample, the left temple bar 106 in the proximity of the left hinge 104(or inside the end cover 108 or 109) as a power source required for theoperation of the eyeglass-type wearable device 100.

A left camera 13L is attached to the end of the left eye frame 102 nearthe left hinge 104, and a right camera 13R is attached to the end of theright eye frame 101 near the right hinge 105. A micro CCD image sensorcan be used for the cameras.

The cameras (13L and 13R) may be used as a stereo camera. Or, aninfrared camera (13R) and a laser (13L) may be provided with the camerapositions as a distance sensor using a combination of the infraredcamera and the laser. The distance sensor may be composed of a microsemiconductor microphone (13R) which collects ultrasonic waves and amicro piezoelectric speaker (13L) which generates ultrasonic waves.

Note that, a center camera (not shown) may be provided with the bridge103 instead of or in addition to the right and left cameras 13R and 13L.Or, the device may not include any camera at all. (The cameras are shownas a camera 13 in FIG. 7.)

A left display 12L is fit in the left eye frame 102, and a right display12R is fit in the right eye frame 101. The display is provided with atleast one of the right and left eye frames and is formed of film liquidcrystal or the like. Specifically, a film liquid crystal display deviceadopting polymer diffusion liquid crystal (PDLC) without a polarizer canbe used as one or both of the right and left displays 12R and 12L (thedisplay is depicted as a display 12 in FIG. 7). Note that, if thedisplay 12R alone is provided with the right eye frame 101, atransparent plastic plate is fit in the left eye frame 102.

The bridge 103 is connected to a transmitter electrode 140 and thetransmitter electrode 140 is electrically and mechanically connected tothe eye frame 101 (and 102). Four receiver electrodes 141 to 144 areprovided with the periphery of the right eye frame 101.

Specifically, a north receiver electrode (upper electrode) 141 isdisposed at the upper side of the right eye frame 101 via (i.e.,insulated from the transmitter electrode) a dielectric layer which isnot shown. Similarly, a south receiver electrode (lower electrode) 142is disposed at the lower side of the right eye frame 101, a westreceiver electrode (right electrode) 143 is disposed at the right sideof the same, and an east receiver electrode (left electrode) 144 isdisposed at the left side of the same. (Generally speaking, the metalbridge 103 which is connected to the transmitter electrode 140 iselectrically connected to the entirety of the metal eye frame 101 andthe electrodes 141 to 144 face the four parts of the eye frame 101through a dielectric insulating layer.) The electrodes 140 to 144 areelectrically separated from each other and are connected to the dataprocessor 11 through insulating interconnection members (not shown). Theelectrodes 140 to 144 are used as capacitance sensors and are structuralcomponents of a gesture detector 14 shown in FIG. 7.

Note that, the electrodes 141 to 144 are depicted conspicuously in FIG.1 for easier understanding. However, in an actual product, theelectrodes 141 to 144 can be formed more inconspicuously by, forinstance, embedding in the eye frames.

Furthermore, capacitance sensor electrodes (141 to 144) are providedwith only the right eye frame 101 side in FIG. 1; however, similarelectrodes (141* to 144*) may be provided with the left eye frame 102side as in the example shown in FIG. 3. In other words, the capacitancesensor electrodes (141 to 144/141* to 144*) can be provided with theright eye frame 101 side and/or the left eye frame 102 side.

A nose pad is disposed between the right and left eye frames 101 and 102and below the bridge 103. The nose pad includes a left nose pad 150L anda right nose pad 150R. Although this is not depicted in FIG. 1, rightnose pad electrodes 151 a and 151 b are provided with the right nose pad150R, and left nose pad electrodes 152 a and 152 b are provided with theleft nose pad 150L (cf. FIGS. 3 to 6).

The electrodes 151 a, 151 b, 152 a, and 152 b are electrically separatedfrom each other and are connected to three AD converters (ADC 1510,1520, and 1512) via insulating interconnection members (not shown).Outputs from the ADCs have different signal waveforms corresponding tomotions of user's eyes adjacent to the right and left eye frames and aresupplied to the data processor 11 in FIG. 7 as digital data withcontents corresponding to the eye motions of the user. The electrodes151 a, 151 b, 152 a, and 152 b are used as sightline detection sensors,and the electrodes 151 a, 151 b, 152 a, and 152 b and three ADconverters are components of an eye motion detector 15 of FIG. 7.

The eyeglass-type wearable device 100 of FIG. 1 is mounted on the headof the user (not shown) by the right and left nose pads (150R and 150L),right and left temple bars (106 and 107), and right and left end covers(108 and 109). In the example of FIG. 1, only the right and left nosepads (150R and 150L), right and left temple bars (106 and 107), andright and left end covers (108 and 109) are in direct contact with thehead (or face) of the user; however, parts other than the above (nosepads, temple bars, and end covers) may be in contact with the user for,for example, balancing a voltage between the ADCs (FIGS. 3, 4, and 5)and the body of the user.

FIG. 2 shows how to obtain detection voltage signals (Vrxbuf) from achange in a capacitance (Ch) corresponding to a gesture (for example, ahand or finger movement of the user). Here, the body of the user whowears the eyeglass-type wearable device 100 in FIG. 1 is at a groundpotential (GND). Since a human body is electrically conductive, thehands and fingers of the user are assumed to be the ground potential(GND). The following explanation will be given as a general example ofhow to obtain the detection signals corresponding to a gesture, in whichthe electrodes 141 to 144 are at one of the right and left eye framesfor simplification.

Here, one receiver electrode (one of 141 to 144, e.g., 141) is betweenthe transmitter electrode 140 and the GND (a hand or finger of the user,for example) and a capacitance between the transmitter electrode 140 andthe receiver electrode 141 is Crxtx. Furthermore, a capacitance betweenthe transmitter electrode 140 and the GND is Ctxg, a capacitance betweenthe receiver electrode 141 and the GND is Crxg, and a capacitancebetween the hand or finger of the user (GND) which performs a gesture tobe detected and the receiver electrode is Ch (Ch varies corresponding toa gesture of the user). In consideration of the capacitance Ch made bythe hand of the user, Crxg+Ch is the total capacitance between thereceiver electrode 141 and the GND. When a high-frequency voltage Vtx isapplied between the transmitter electrode 140 and the GND, the signalvoltage obtained from the receiver electrode 141 will be expressed asfollows.

Vrxbuf=Vtx×{(Crxtx)/(Crxtx+Crxg+Ch)}   (1)

The capacitances (Crxtx and Crxg) are different in each of the receiverelectrodes 141 to 144, and the capacitance (Ch) varying corresponding tothe gesture of the user is different in each of the receiver electrodes141 to 144. Therefore, the voltage signals (Vrxbuf1 to Vrxbuf4) obtainedfrom respective receiver electrodes 141 to 144 will be different.However, each of the different voltage signals (Vrxbuf1 to Vrxbuf4) canbe obtained by the formula (1).

From the four receiver electrodes 141 to 144, four voltage signals(Vrxbuf1 to Vrxbuf4) each varying corresponding to the gesture of theuser can be obtained. A change manner in the voltage signals correspondsto a gesture of the user (for example, if the four voltage signals arerepresented by bar graphs, the heights of the four bars are independentand different from each other but a pattern of changes in the fourbar-heights should correspond to the gesture of the user). The fourvoltage signals (Vrxbuf1 to Vrxbuf4) change corresponding to themovements of a hand or a finger such as up-and-down and right-to-leftswings, clockwise or counterclockwise rotations, and movements closer toor distant from the receiver electrodes. Thus, if correspondingrelationships between the gesture patterns of users (hand or fingerup-and-down movement, rotation, and the like) and change patterns of thefour voltage signals (Vrxbuf1 to Vrxbuf4) are checked or examined inadvance, the gestures of users can be identified and detected.Consequently, a gesture of swiping up a finger from the below (southside) to the above (north side) can be translated into a command ofscreen scroll from the below to the above, for example.

Note that, a 3D gesture sensor using the formula (1) is commerciallyavailable as MGC3130 (Single-Zone 3D Tracking and Gesture Controller) ofMicrochip Technology Inc. and its detailed data sheet can be obtainedthrough the Internet. The principle of the 3D gesture sensor using theformula (1) is a publically-known technique. However, the embodiment inwhich a combination of the 3D gesture sensor and an eye motion sensor isused with an AR display by images IM1/IM2 (cf. FIG. 3) should be novel.(Here, “AR” is an acronym of Augmented Reality and indicates atechnology of adding information to the real world viewed throughglasses, for example.)

FIG. 3 shows an eyeglass-type wearable device of another embodiment, andshows an example of the arrangement of capacitance sensor electrodes(140 to 144 and 141* to 144*) for the gesture detection and an exampleof the arrangement of eye motion detection electrodes (151 a, 151 b, 152a, and 152 b) provided with a nose pad. In the example of FIG. 3,receiver electrodes (141 to 144 and 141* to 144*) functioning the sameas the receiver electrodes 141 to 144 depicted relatively large in FIG.1 are arranged in the periphery of the eye frames 101 and 102 in aninconspicuous manner. (The receiver electrodes 141 to 144 and thereceiver electrodes 141* to 144* in FIG. 3 may, with slightexaggeration, be symmetrically arranged at right and left sides with apositional relationship similar to the electrodes 141 to 144 in FIG. 1.)

In FIG. 3, the receiver electrodes 141 to 144 at the right side areinsulated from each other, and are disposed to face the metal part ofthe frame 101 connected to the transmitter electrode 140 via aninsulating material (such as a plastic or a polypropylene film oftenused in a small capacitor) which is not shown. Similarly, the receiverelectrodes 141* to 144* at the left side are insulated from each other,and are disposed to face the metal part of the frame 102 connected tothe transmitter electrode 140 via an insulating material which is notshown.

In FIG. 3, right nose pad electrodes 151 a and 151 b are disposed aboveand below the right nose pad 150R, and left nose pad electrodes 152 aand 152 b are disposed above and below the left nose pad 150L. Outputsfrom the right nose pad electrodes 151 a and 151 b are supplied to theADC 1510, and outputs from the left nose pad electrodes 152 a and 152 bare supplied to the ADC 1520, and outputs from the lower right and leftnose pad electrodes 151 b and 152 b (or outputs from the upper right andleft nose pad electrodes 151 a and 152 a) are supplied to the ADC 1512.

Ch1 signals which change corresponding to up-and-down motions of theright eye of the user can be obtained through the ADC 1510. Ch2 signalswhich change corresponding to up-and-down motions of the left eye of theuser can be obtained through the ADC 1520. Ch0 signals which changecorresponding to motions of the right and left eyes of the user can beobtained through the ADC 1512. The up-and-down motions of the right andleft eyes of the user can be evaluated by Ch1+2 signals representing anaverage of outputs of the ADCs 1510 and 1520. (A relationship betweensignal waveforms of Ch0, Ch1, Ch2, and Ch1+2 and eye motions will bedescribed later with reference to FIGS. 8 to 14.)

Film liquid crystal of the right display 12R in FIG. 3 can display aright display image IM1 including, e.g., an icon group of a ten-keys(numbers, operators, enter-key, and the like), alphabets, and the like.Film liquid crystal of the left display 12L can display a left displayimage IM2 including, e.g., optional character strings, icons, and thelike (contents shown on the displays 12R and 12L are optional). Ten-keysand alphabets shown on the right display 12R (or on the left display12L) may be used for the input of numbers and letters. Character stringsand icons displayed on the right display 12R (or on the left display12L) may be used for the retrieval of specific data items and theselection/determination of a target item.

The display images IM1 and IM2 can be used to provide the augmentedreality (AR) in which data including numbers and letters is added to thereal world viewed through the glasses. The contents of the display imageIM1 and the contents of the display image IM2 can be the same (IM1=IM2)or different (IM1≠IM2) depending on the type of embodiments.Furthermore, the display image IM1 (or IM2) can be displayed in theright display 12R and/or the left display 12L. If the contents of the ARdisplay are required to be shown in a 3D image (with a depth)overlapping the real world viewed through the glasses, the displayimages IM1 and IM2 are different images for 3D display.

Furthermore, if the displays (12R and 12L) are positioned right andleft, the images on the right and left displays (IM1 and IM2) can beshifted in opposite directions by, for example, adjusting an angle ofconvergence. This will reduce the workload of eyes viewing a target inthe real world and the AR display alternately. However, normally, thesame images are displayed in the right and left displays (12R and 12L).

The display control of the displays 12R and 12L can be performed by thedata processor 11 embedded in the right temple bar 107. (Displayingletters and icons on a display is a well-known technique.) Powerrequired for the operation of the data processor 11 and the like can beobtained from a battery BAT embedded in the left temple bar 106.

Note that, if a designer may wear a test product corresponding to theexample of FIG. 3 and feel that a weight balance of the product isinappropriate, one of the reasons causing such inappropriateness may bethe battery BAT in the left temple bar 106. In that case, a sinker maybe provided with the right temple bar 107 to balance with the batteryBAT in the left temple bar 106.

As in the example of FIG. 3, if the sensor electrodes (141 to 144 and141* to 144*) are provided with the both sides of the device while thedata processor 11 is provided with one side, a very-small flat cable(not shown) is passed through the frames 101 and 102 inconspicuouslysuch that the electrodes 141* to 144* at the left side are connected tothe data processor 11 at the right side. Similarly, a very-small flatcable (not shown) is passed through the frame 101 inconspicuously suchthat the electrodes 141 to 144 at the right side are connected to thedata processor 11 at the right side. A similar very-small flat cable maybe used in the connection of the nose pad electrode (151 a, 151 b, 152a, and 152 b) to the data processor 11.

If two pairs of capacitance sensor electrodes (140 to 144 and 141* to144*) for the gesture detection are disposed at both right and leftsides, the number of the receiver electrodes of capacitance sensor iseight in total at the both sides. Then, eight kinds of detection signals(Vrxbuf) each changing corresponding to 3D gestures of right and lefthands (or two or more fingers) are obtained. Data input A (FIG. 7) canbe generated by combinations of changes in the detection signals.Various gestures can be detected using the data input A (for example,several sign language patterns may be detected).

Furthermore, with the two pairs of capacitance sensor electrodes (140 to144 and 141* to 144*) for the gesture detection disposed at both rightand left sides, a detectable range of the gesture movement (especiallyin the horizontal direction) can be increased. For example, in theexample of FIG. 3, five gesture sections (right end of the right eyeframe 101, center of the right eye frame 101, center of the bridge 103,center of the left eye frame 102, and left end of the left eye frame102) will be given. In that case, fingers of a right hand can be movedbetween the right end of the right eye frame and the center of the righteye frame, between the right end of the right eye frame and the centerof the bridge, between the right end of the right eye frame and thecenter of the left eye frame, and between the right end of the right eyeframe and the left end of the left eye frame (or to the outside of theleft end of the left eye frame).

A section in which a gesture is performed in the five sections can bedetermined based on a change condition of eight signal levels from theeight receiver electrodes of the capacitance sensors. (For example, if afinger is swung from right to left between the right end of the righteye frame to the left end of the left eye frame, eight electrode signallevels all change individually.) With the gesture movable range dividedas above, a section in which a gesture is performed can be identifiedeven if gestures in the same pattern are performed in any sections.Thus, determination results as to the sections in which the gestures areperformed can be used to substantially increase the types of thecommands input by data input A (as compared to a case where movablerange is not identified).

Note that, in the example of FIG. 3 (or FIG. 1), a 3D gesture isdetected when a right-handed user uses his/her right hand (rightfingers) for the gesture in a 3D space in right front of the user (aspace where the user sees the display image IM1) with the electrodes 141to 144 at the right eye frame 101 side. Furthermore, as in the exampleof FIG. 3, with the capacitance sensor electrodes 141* to 144* for thegesture detection provided with the left eye frame 102 side (to surroundthe left display image IM2), a 3D gesture by a left hand in the 3D spacein the left front of the user can be detected for improving theoperability of a left-hand user.

If the device is made for a left-hand user only, only the electrodes141* to 144* at the left eye frame 102 side may be used as thecapacitance sensors for the gesture detection, and only the displayimage IM2 may be used for the gesture operation. That is, the electrodes141 to 144 at the right eye frame 101 side and the display image IM1 maybe omitted from a certain embodiment (the display contents of thedisplay image IM2 may be the same as or different from the contents tobe displayed by the display image IM1).

FIG. 4 shows an eyeglass-type wearable device of a still anotherembodiment. In this example, electrodes (140 to 144) of a capacitancesensor 14 for the gesture detection are provided with the right templebar 107 side and left electrodes (140* to 144*) of a capacitance sensor14* for the gesture detection are provided with the left temple bar 106side. The right face of a user contacting the right temple bar 107 andthe left right face of the user contacting the left temple bar 106 arethe GND. A plastic tab 14T on which the electrodes 140 to 144 of thecapacitance sensor 14 are formed to be electrically insulated from theGND is attached to the right temple bar 107. Similarly, a plastic tab14T* on which electrodes 140* to 144* of the capacitance sensor 14T* areformed to be electrically insulated from the GND is attached to the lefttemple bar 106.

Tabs may be attached to the temple bars through the following manners,for example. That is, the tab 14T (or 14T*) may be mechanically fixed tothe temple bar 107 (or 106) undetachably. Or, the tab 14T (or 14T*) maybe detachably attached to a connector receiver (not shown) provided withthe temple bar 107 (or 106) using a snap-lock multipoint connector orthe like. A connector which detachable attaches the tab and the templebar may be a micro USB or a micro HDMI (registered trademark) inconsideration of a mechanical design for the sufficient mechanicalstrength after the connection.

In the example of FIG. 4, data processors 11 and 11* having the samefunctions are disposed inside the right temple bar 107 and the lefttemple bar 106, respectively. Furthermore, a battery BAT is attachedinside the thick part of the right end cover 109, and a battery BAT* isattached inside the thick part of the left end cover 108.

In the example of FIG. 4, the right and left temple bars 107 and 106 aremounted partly on the rear sides of the right and left ear tabs (notshown) to be put on the head of the user. In that case, if the upperends of the rear sides of the ear tabs of the user are considered asfulcrums, the weight balance between the front parts of the fulcrums(the part of the eye frames 101 and 102) and the rear parts of thefulcrums (the part of the end covers 109 and 108) is improved by theweight of the BAT and BAT*.

Furthermore, since the BAT and BAT* are arranged at the right and leftsides, the right and left weight balance of the eyeglass-type wearabledevice 100 can be improved as being viewed from the center of the rightand left eyes of the user.

Note that, although this is not shown, the structure of two dataprocessors 11 and 11* provided with the right and left temple bars 107and 106 and/or the structure of the two batteries BAT and BAT* providedwith the right and left end covers 109 and 108 can be applied to theexample of FIG. 3.

In the example of FIG. 4, representative gestures of a user may befrontward-and-backward and up-and-down movements of a hand or fingers inthe proximity of the plastic tab 14T (or 14T*), rotation of the hand andfingers in the proximity of sides of the face, and movements to put thehand and fingers near to or away from the face.

FIGS. 5(a) to 5(e) show various examples of the nose pad electrodes (151a, 151 b, 152 a, and 152 b) for the eye motion detection provided withthe nose pads (150R and 105L). FIG. 5(a) shows four nose pad electrodes151 a, 151 b, 152 a, and 152 b provided with the right and left nosepads in a vertically and horizontally symmetrical manner.

FIG. 5(b) shows an example where the four nose pad electrodes 151 a, 151b, 152 a, and 152 b are provide with the right and left nose pads in ahorizontally symmetry but vertically asymmetry manner. A down pressingforce caused by the weight of the right and left eye frames works on thenose pads (150R and 150L). Thus, the lower nose pad electrodes (151 band 152 b) sufficiently contact the skin of the nose of the user even ifthe area of the electrodes is small while the upper nose pad electrodes(151 a and 152 a) may not contact well with the skin of the nose of theuser.

Even if the nose pads (150R and 150L) are pressed down by the weight ofthe right and left eye frames and the contact of the upper nose padelectrodes (151 a and 152 a) tend to be insufficient, such insufficientcontact of the upper nose pad electrodes (151 a and 152 a) can beimproved by increasing the area of the upper nose pad electrodes (151 aand 152 a) as in FIG. 5(b).

FIG. 5(c) shows an example where the four nose pad electrodes 151 a, 151b, 152 a, and 152 b are provide with the right and left nose pads in ahorizontally and vertically asymmetry manner. The arrangement of FIG.5(c) can be obtained through about 180° rotation of one of the nose padsof FIG. 5(b) (150R in this example). Depending on a skin condition ofthe nose of the user, location or posture of the user, or a mountcondition of the glasses, better contact of the right and left nose padelectrodes may be obtained in the example of FIG. 5(c) than FIG. 5(b).In such a case, the right and left nose pads (150R and 150L) may be maderotatable such that both the arrangements of FIGS. 5(b) and 5(c) can beselected by the user.

The electrodes 151 a, 151 b, 152 a, and 152 b of FIGS. 5(a) to 5(c) areprepared by, for example, performing a metal evaporation process of apredetermined electrode pattern, printing a conductive paint, orattaching an electrode piece on a nose pad material of an insulatingmaterial/dielectric (such as ceramic, plastic, and rubber) formed in apredetermined shape. The electrodes 151 a, 151 b, 152 a, and 152 b maybe flushed with the surface of the nose pad material, or may be formedas bumps on the surface of the nose pad material.

In the examples of FIGS. 5(d) and 5(e), holes are pierced throughcertain points on the right and left nose pads 150R and 150L and smallmetal rings are put in the holes to attach the four nose pad electrodes151 a, 151 b, 152 a, and 152 b. In the examples, ring-shaped nose padelectrodes 151 a, 151 b, 152 a, and 152 b are shown; however, nolimitation is intended thereby. These nose pad electrodes may bepolygonal with rounded corners or may be partly cut such as a letter C,for example.

FIG. 6 shows an example of how to extract detection signals from the eyemotion detection electrodes (151 a, 151 b, 152 a, and 152 b) providedwith the nose pads. A potential difference between the upper electrode151 a and the lower electrode 151 b of the right nose pad 150R isreceived by high input impedance of the ADC 1510 and Ch1 potentialdifference between the upper and lower electrodes which may vary withtime is detected as digital data. A potential difference between theupper electrode 152 a and the lower electrode 152 b of the left nose pad150L is received by high input impedance of the ADC 1520 and Ch2potential difference between the upper and lower electrodes which mayvary with time is detected as digital data.

Furthermore, a potential difference between the lower electrode 152 b ofthe left nose pad 150L and the lower electrode 151 b of the right nosepad 150R is received by high input impedance of the ADC 1512 and Ch0potential difference between the right and left electrodes which mayvary with time is detected as digital data. (Or, a potential differencebetween the upper electrode 152 a of the left nose pad 150L and upperelectrode 151 a of the right nose pad 150R may be received by high inputimpedance of the ADC 1512 and Ch0 potential difference between the rightand left electrodes which may vary with time may be detected as digitaldata.)

Note that ADCs 1510, 1520, and 1512 of FIG. 6 may be an ADC having aworking voltage Vdd=3.3 V and a resolution of 24 bit. In that case, theweight of the detection signal level is 3.3 V/(2{circumflex over( )}24)=(nearly) 200 nV. In the detection signal level shown in FIGS. 8to 14, if the amplitude values of Ch1 and Ch2 are represented by 1000,for instance, the detection signal level from the ADCs is approximately200 μV in voltage.

Types of the eye motion and ranges of eye motion related to the eyemotion detection of FIG. 6 are, for example, as follows.

<Types of Eye Motion>

(01) Compensative eye motion

Non-voluntary eye motion developed for stabilizing an external image ona retina regardless of motions of the head or body.

(02) Voluntary eye motion

Eye motion developed to set a target image to the center of the retinaand controlled voluntarily.

(03) Impulsive eye motion (Saccade)

Eye motion made when a focus point is changed to see an object (easy todetect).

(04) Slide eye motion

Smooth eye motion made when tailing an object moving slowly (hard todetect).

<Motion range of eyes (of an ordinary adult)>

(11) Horizontal directions

Left direction: 50° or less

Right direction: 50° or less

(12) Vertical directions

Lower direction: 50° or less

Upper direction: 30° or less

(The range of angles voluntarily movable in the vertical directions isnarrower in the upper direction. Since the Bell phenomenon in which eyerotate upward when eyes are closed, the eye motion range in the verticaldirections shifts to the upper direction when the eyes are closed.)

(13) Others

Angle of convergence: 20° or less

FIG. 7 shows the data processor 11 attachable to the eyeglass-typewearable devices of various embodiments and peripheral devices. In theexample of FIG. 7, the data processor 11 includes a processor 11 a,nonvolatile memory 11 b, main memory 11 c, and communication processor11 d, for example. The processor 11 a is a microcomputer having acomputing performance corresponding to a product specification. Variousprograms executed by the microcomputer and various parameters used inthe program execution can be stored in the nonvolatile memory 11 b. Thework area to execute the programs can be provided by the main memory 11c.

Commands to be executed by the processor 11 can be obtained via thecommunication processor 11 d from an external server (or a personalcomputer) which is not shown.

The communication processor 11 d can use available communication schemessuch as ZigBee (registered trademark), Bluetooth (registered trademark),and Wi-Fi (registered trademark). A process result from the processor 11a can be sent to the storage management server or the like through thecommunication processor 11 d.

A system bus of the data processor 11 is connected to a display 12 (12Rand 12L of FIGS. 1, 3, and 4), camera 13 (13R and 13L of FIG. 1),gesture detector 14, and eye motion detector 15. Power is supplied toeach device (11 to 15) of FIG. 7 by a battery BAT.

The gesture detector 14 of FIG. 7 includes the electrodes 140 to 144 ofcapacitance sensors, and circuits to output data based on a changepattern of the above-described four voltage signals (Vrxbuf1 to Vrxbuf4)to the processor 11 a. From the change pattern (for example,corresponding to a swiping up motion of a finger) of the four voltagesignals (Vrxbuf1 to Vrxbuf4), the processor 11 a interprets a commandcorresponding to the gesture of the user (for example, a command toscroll up the character strings in the image IM2 displayed on thedisplay 12L of FIG. 3), and executes the upward scroll in the display12.

The command is an example of data input A using the gesture detector 14.

The eye motion detector 15 of FIG. 7 includes four eye motion detectionelectrodes (151 a, 151 b, 152 a, and 152 b) which are components of thesightline detection sensor, three ADCs (1510, 1520, and 1512) whichextract digital signals corresponding to eye motions from theelectrodes, and circuits to output the output data (data correspondingto detection signal waveforms of FIGS. 8 to 14) from ADCs to theprocessor 11 a. From various eye motions (up-and-down, right-and-left,blinks, closed eyes, and the like) of the user, the processor 11 ainterprets a command corresponding to the eye motion type and executesthe command.

Specific commands corresponding to the types of eye motions may be, forexample, selecting a data item in the line of sight if the eye motion isclosing eyes (similar to a click of a computer mouse), starting aprocess of the selected data item if the eye motion is continuous blinksor a wink (similar to double clicks of a computer mouse). The command isan example of data input B using the eye motion detector 15.

Now, a method of detecting (estimating) an eyesight direction of a userwill be explained. FIG. 8 shows an electro-oculogram (EOG) with respectto a relationship between an eye motion from the front to the above anddetection signal levels (Ch0, Ch1, Ch2, and average level Ch1+2 of Ch1and Ch2) obtained from ADCs (1510, 1520, and 1512) of FIG. 6. The eyemotion detection is performed based on the detection signal waveforms inthe broken-line frame in the figure. The reference of the detection is acase where there is no eye motion when a user seeing the direct front(in that case, a condition of left outside of the broken line frame ofFIG. 8, and output signal waveforms Ch0 to Ch2 from the three ADCs ofFIG. 6 are substantially flat while the user staring the direct frontwithout blinking and there is almost no change through time).

The user sees the direct front with his/her both eyes, instantly movesthe sight upward and maintain the upward stare for one second, and theninstantly returns the stare in the front. This is repeated for fivetimes and changes of the detection signal levels are shown in FIG. 8.

FIG. 9 shows an eye motion detection similar to that of FIG. 8 when thesight moves from the front to the below. From the waveform changes ofFIGS. 8 and 9, whether the sight is upward or whether the sight isdownward can be detected using the case where the sight is in front as areference.

FIG. 10 shows an electro-oculogram (EOG) with respect to a relationshipbetween an eye motion from the left to the right and detection signallevels (Ch0, Ch1, Ch2, and average level Ch1+2 of Ch1 and Ch2) obtainedfrom the three ADCs of FIG. 6. With the eye motion from the left to theright, the change of the detection signal waveform of Ch0 through timegoes up to the right side (although this is not shown, with the eyemotion from the right to the left, the change of the detection signalwaveform of Ch0 through time goes down to the right side). From thewaveform changes of Ch0, whether the sight is rightward or whether thesight is leftward can be detected using the case where the sight is infront as a reference.

If the detection results of FIGS. 8 to 10 are combined, it can be knownthat which direction the sight points to the up, down, right, and leftdirections, using the case where the sight is in front as a reference.

FIG. 11 shows an electro-oculogram (EOG) with respect to a relationshipbetween an eye motion repeating blinks (both eyes) for five times withfive second intervals and detection signal levels (Ch0, Ch1, and Ch2)obtained from the three ADCs of FIG. 6. Blinks of both eyes can bedetected by pulses in Ch1 and Ch2. Blinks unconsciously performed by auser do not have a periodicity in most cases.

Therefore, by detecting a plurality of pulses with certain (roughlyconstant) intervals as shown in FIG. 11, intentional blinks of a usercan be detected. (Generally speaking, one blink motion takes 100 to 150msec and the sight is blocked by a blink motion for approximately 300msec.)

FIG. 12 shows an electro-oculogram (EOG) with respect to a relationshipbetween an eye motion repeating blinks (both eyes) of an eye closing forone second and an eye opening for four seconds for five times anddetection signal levels (Ch0, Ch1, and Ch2) obtained from the three ADCsof FIG. 6. Closing both eyes can be detected by a wide pulse in Ch1 andCh2 (if eyes are closed intentionally, it takes longer than a blink andthe pulse width detected becomes wider). By detecting the wide pulses ofCh1 and Ch2 shown in FIG. 12, the intentional eye closing of the usercan be detected.

Note that, although this is not shown, a wide pulse shows in Ch1 whenthe user closes the right eye only and a wide pulse shows in Ch2 whenthe user closes the left eye only. Thus, a right eye closing and a lefteye closing can be detected separately.

FIG. 13 shows an electro-oculogram (EOG) with respect to a relationshipbetween an eye motion repeating blinks (both eyes) for five times andrepeating left eye winks (blinks of left eye) for five times with eyesfront and detection signal levels (Ch0, Ch1, and Ch2) obtained from thethree ADCs of FIG. 6.

As shown in FIG. 6, the position of the ADC 1512 of Ch0 is offset lowerthan a center line of the right and left eyeballs. Because of theoffset, negative direction potential changes appear in both + input and− input of the ADC 1512 of FIG. 6 when both eyes blink. Then, if thepotential changes (amount and direction) of both+ input and − input aresubstantially the same, these changes are almost canceled and the signallevel output from the ADC 1512 of Ch0 may be substantially constant (cf.Ch0 level in a left broken line frame of FIG. 13). On the other hand,one eye (left eye) blink does not substantially change the potential atthe − input side of the ADC 1512 and a relatively large negativedirection potential change appears at the + input side of the ADC 1512.Then, a cancel amount of the potential changes between + input and −input of the ADC 1512 is reduced and a small pulse (small wave in thesignal level) appears in the negative direction in the signal levelsoutput from the ADC 1512 of Ch0 (cf. Ch0 level in a right broken lineframe of FIG. 13). From the polarity of the small wave in the signallevel (pulse in the negative direction), a left eye wink can be detected(an example of left wink detection using Ch0).

Note that, if the potential change of the + input and − input of the ADC1512 cannot be set even because of the distortion of the face of theuser or the skin condition, a calibration to set the output of the ADCof Ch0, detected when the user wears the eyeglass-type wearable device100 and brinks both eyes, to minimum (to set a cancel amount between +input components and − input components maximum) should be performed inadvance.

Furthermore, if a peak ratio SL1 a/SL2 a of the detection signalsCh1/Ch2 at the time of a both eye wink is used as a reference, a peakratio SL1 b/SL2 b at the time of a left eye wink changes (SL1 b/SL2 b isnot equal to SL1 a/SL2 a). From this point, a left wink can be detected.

FIG. 14 shows an electro-oculogram (EOG) with respect to a relationshipbetween an eye motion repeating blinks (both eyes) for five times andrepeating right eye winks (blinks of right eye) for five times with eyesfront and detection signal levels (Ch0, Ch1, and Ch2) obtained from thethree ADCs of FIG. 6.

As stated above, the position of the ADC 1512 of FIG. 6 is offset lowerthan a center line of the right and left eyeballs. Because of theoffset, negative direction potential changes appear in both+ input and −input of the ADC 1512 of FIG. 6 when both eyes blink. Then, if thepotential changes (amount and direction) of both+ input and − input aresubstantially the same, these changes are almost canceled and the signallevel output from the ADC 1512 of Ch0 may be substantially constant (cf.Ch0 level in a left broken line frame of FIG. 14). On the other hand,one eye (right eye) blink does not substantially change the potential atthe + input side of the ADC 1512 and a relatively large negativedirection potential change appears at the − input side of the ADC 1512.Then, a cancel amount of the potential changes between + input and −input of the ADC 1512 is reduced and a small pulse (small wave in thesignal level) appears in the positive direction in the signal levelsoutput from the ADC 1512 of Ch0 (cf. Ch0 level in a right broken lineframe of FIG. 14). From the polarity of the small wave in the signallevel (pulse in the negative direction), a right eye wink can bedetected (an example of right wink detection using Ch0).

Furthermore, if a peak ratio SR1 a/SR2 a of the detection signalsCh1/Ch2 at the time of a both eye wink is used as a reference, a peakratio SR1 b/SR2 b at the time of a right eye wink changes (SR1 b/SR2 bis not equal to SR1 a/SR2 a). Furthermore, the peak ratio SL1 b/SL2 b ofa left wink and the peak ratio SR1 b/SR2 b of a right wink may bedifferent (how different they are can be confirmed by an experiment).

From this point, a right wink can be detected separately from the leftwink (an example of right and left wink detections using Ch1 and Ch2).

Using Ch0 or Ch1/Ch2 for detecting the right and left winks can bearbitrarily determined by a device designer. Results of right and leftwink detections using Ch0 to Ch2 can be used as operation commands.

FIG. 15 is a flowchart which shows processes performed by combinationsof gesture data inputs (data input A) and eye motion data inputs (datainput B) when the eyeglass-type wearable device of FIG. 3 is used, forexample.

For example, the eyeglass-type wearable device 100 of FIG. 3 with thedata processor 11 of FIG. 7 is wirelessly connected to a server (notshown).

If an item list related to a plurality of items is sent from a server tothe device 100 through, for example, Wi-Fi, data of the item list arestored in the memory 11 c of FIG. 7. A program executed in the processor11 a displays an image IM1 (or IM2) of at least part of item data fromthe data of the items included in the stored item list on the rightdisplay 12R (or the left display 12L) (ST10 of FIG. 15). The imagedisplay may be performed in the right display 12R in default. However,there may be a user who does not prefer a gesturing finger is seenmoving ahead of the right display, and thus, the image display may beperformed in the left display 12L at which a finger of the right hand isnot easily seen if the user choose so (optionally).

If currently necessary item data (name of the item and an ID codethereof) are not being displayed in the displayed list, the user withthe device 100 moves, for example, his/her right index finger swiping upin front of the right eye frame 12R with the electrodes (141 to 144) ofthe gesture detector 14. Then, the type of the motion (one of thegestures) is determined (ST12), and the data input A corresponding tothe motion is generated in the gesture detector 14 (ST14). The datainput A is sent to the processor 11 a through the system bus of FIG. 7.Then, the program executed in the processor 11 a scrolls up the itemdata in the image IM1 (or IM2) displayed in the right display 12 (or theleft display 12L) (ST16). By repeating the finger swiping up gesture,the item data in the image

IM1 (or IM2) can be scrolled up to the end.

If desired item data are not found through the scroll, the right indexfinger, for example, is swiped down. The type of the motion (one of thegestures) is determined (ST12), and data input A corresponding to themotion is generated in the gesture detector 14 (ST14). The data input Ais sent to the processor 11 a, and the item data in the image IM1 (orIM2) displayed in the right display 12R (or in the left display 12L) arescrolled downward (ST16). By repeating the finger swiping down gesture,the item data in the image IM1 (or IM2) can be scrolled down to the end.

If a plurality of item lists are displayed in the image IM (or IM2), theitem list seen by the user can be detected by the sightline detectionsensor of the eye motion detector 15. Now, for a simplified explanation,a case where three item data lines (upper, middle, and lower lines) aredisplayed in the image IM1 (or IM2) is given.

When the user stares in front and stays still, signal waveforms of thethree ADCs (Ch0 to Ch2) of FIG. 6 are all substantially flat. Then, thesightline of the user is determined to be directed to the middle itemdata displayed in the image IM1 (or IM2) (or the user is estimated tosee the item data in the middle line).

When the user stares in front and looks up, signal waveforms of thethree ADCs (Ch0 to Ch2) of FIG. 6 show upward pulses (FIG. 8). Then, thesightline of the user is determined to be directed to the upper itemdata displayed in the image IM1 (or IM2) (or the user is estimated tosee the item data in the upper line).

When the user stares in front and looks down, signal waveforms of thethree ADCs (Ch0 to Ch2) of FIG. 6 show downward pulses (FIG. 9). Then,the sightline of the user is determined to be directed to the lower itemdata displayed in the image IM1 (or IM2) (or the user is estimated tosee the item data in the lower line).

When the user stares in front and closes both eyes for a short period(0.5 to 1.0 seconds), upward pulses having waveforms different from thatof FIG. 8 show

(FIG. 12). Then, the item data displayed in the center of the image IM1(or IM2) are determined to be selected by the user (similar to one clickby a computer mouse). Similarly, if the user looks up and closes botheyes, the item data in the upper line are determined to be selected, andif the user looks down and closes both eyes, the item data in the lowerline are determined to be selected.

After the selection of the item data, if the user looks in front andinstant blinks (0.2 to 0.3 seconds) for a few times by both eyes, a fewsharp pulses occur (FIG. 11).

Then, the selection of the item data displayed in the center of theimage IM1 (or IM2) is determined to be decided by the user (similar todouble clicks by a computer mouse).

Similarly, if the user looks up and blinks for a few times by both eyes,the selection of the item data in the upper line is determined to bedecided, while if the user looks down and blinks for a few times by botheyes, the selection of the item data in the lower line is determined tobe decided.

After the selection of the item data, if a left wink is performed (FIG.13), an operation corresponding to the wink can be performed. Forexample, if the user looks in front and winks the left eye, a cursor(not shown) in the character strings of the item data displayed in thecenter of the image IM1 (or IM2) can be moved to the left. Conversely,if a right wink is performed, the cursor (not shown) in the characterstrings of the item data displayed in the center of the image IM1 (orIM2) can be moved to the right.

As can be understood from the above, the eye motions of the userincluding the eye direction of the user (up-and-down and right-and-leftmotions, blinks, closed eyes, winks, and the like) can be determinedusing combination of various signal waveforms obtained from thesightline detection sensor of the eye motion detector 15 (ST22).

After the determination of the eye motion of the user including the eyedirection of the user (ST22), a data input B corresponding adetermination result is generated by the eye motion detector 15 (ST 24).The data input B is sent to the processor 11 a, and the processor 11 aperforms the process corresponding to the data input B (ST26). Forexample, the processor 11 a determines that an item (not shown)corresponding to the selected item data is picked up by the user fromthe storage rack in the warehouse, and modifies the item list stored inthe memory 11 c. Then, the modified list is informed to the server (notshown) through Wi-Fi (ST26). Or, the user can add a desired value codeor the like to the selected item data using a ten-key in the image IM1(or IM2) displayed in the right display 12R (or left display 12L) ofFIG. 3, for example (ST26).

The process of FIG. 15 is repeated while either the process based ondata input A or the process based on data input B is performed (NO inST28). The process of FIG. 15 is terminated if both the process based ondata input A and the process based on data input B are finished

(YES in ST28).

Steps ST12 to ST16 of FIG. 15 (the process based on data input A) areperformed by a gesture of the user (for example, hand or finger motion)and steps ST22 to ST26 (the process based on data input B) are performedby an eye motion of the user. The process of the data input A and theprocess of the data input B are in cooperation but independent asoperations of the user. Therefore, eye strain is small as compared to acase where the data input is performed by the eye motion only. On theother hand, if a gesture input cannot be performed when using bothhands, data input by eye motion only can be performed.

Furthermore, the eyeglasses-type wearable device 100 of the embodimentscan be operated without touching by hands, and even if fingers aredirty, data input can be performed without dirtying the device 100.

Note that the device may be structured such that the user can touch anyof the electrodes 141 to 144 (with clean fingers). In that case, thecapacitance sensor 14 can be used as a pointing device like a touch pad(a variation of ST12 to ST16 of FIG. 15). For example, in the structureof FIG. 3, a ten-key and a cursor are shown in the display 12R, and thecursor can be moved by touching any of the electrodes 141 to 144 of thecapacitance sensor 14 by a finger. Then, the closed-eyes, blinks, and(right and left) winks detected by the sightline detection sensor 15 areprepared as commands, and a value (character) on which the cursor ispositioned can be selected or decided (entered). As above, using amethod other than a gesture, data input A from the capacitance sensor 14and data input B from the sightline detection sensor 15 can be combinedand various data inputs can be achieved.

In the combination data input operation (combination of data input A anddata input B), an image process of an image taken by a camera or arecognition process of audio caught by a microphone can be unnecessary.Therefore, even in a dark environment unsuitable for a proper imageprocess or in a noisy environment unsuitable for a proper audio input,various data inputs can be performed without touching a specific object.In other words, various data inputs can be performed regardless of thebrightness or the darkness of the operation environment or of the noiseof the operation environment.

Furthermore, the eyeglasses-type wearable device of an embodimentincludes a plurality of eye motion detection electrodes 151 a, 151 b,152 a, and 152 b directly contacting the user, but these electrodes areonly provided with the nose pads (150R and 150L) (the electrodes 140 to144 of the gesture detector do not directly contact the user). Since thenose pads are used in ordinary glasses, the eyeglasses-type wearabledevice of the embodiment can be worn by a person who wears glassesordinarily without feeling uncomfortable. (If a directly-touchingdetection electrode is provided with a part which does notconventionally contact a user such as a bridge part between the rightand left eye frames, some user may feel uncomfortable or may beirritated. However, since the detection electrodes are provided withonly the part which contacts the user in the ordinary glasses (with thenose pads or the temple bars), the eyeglasses-type wearable device ofthe embodiments can be worn without feeling uncomfortable.)

[1] According to an embodiment, an eyeglasses-type wearable device (100in FIGS. 1, 3, and 4) has right and left eye frames (101, 102)corresponding to positions of right and left eyes and nose pads (150R,150L) corresponding to a position of a nose. The eyeglasses-typewearable device includes a display (12R, 12L) provided with at least oneof the right and left eye frames, a gesture detector (14 in FIG. 7)which detects a gesture indicative of a movement of a user, and an eyemotion detector (15 in FIG. 7) which detects eye motion or eye movementof the user.

The eyeglasses-type wearable device performs data input using acombination of a first data input (data input A) corresponding to thegesture detected by the gesture detector and a second data input (datainput B) corresponding to the eye motion detected by the eye motiondetector.

[2] The gesture detector (14 in FIG. 3) comprises a capacitance sensorincluding a plurality of electrodes (141 to 144 and 141* to 144*), andthe electrodes are provided with at least one of the right and left eyeframes (101, 102).

[3] The eyeglasses-type wearable device (100 in FIG. 4) includes rightand left temple bars (106, 107) connected to side ends of the right andleft eye frames (101, 102), respectively. The gesture detector (14 inFIG. 4) comprises one or more tabs (14, 14T) on which a capacitancesensor including a plurality of electrodes (141 to 144 and 141* to 144*)is formed, and the tabs are provided with at least one of the right andleft temple bars (106, 107).

[4] The gesture detector (14) includes a capacitance sensor including aplurality of electrodes (transmitter electrode 140 and upper, lower,right, and left receiver electrodes 141 to 144). A plurality ofcapacitances (Crxtx, Crxg) formed in the electrodes, and a plurality ofelectrode signals (Vrxbuf: electrode signals Vrxbuf1 to vrxbuf4 fromfour respective receiver electrodes) as a function(Vtx*Crxtx/(Crxtx+Crxg+Ch)) of a capacitance

(Ch) which changes depending on a gesture of the user (for example, amovement of a finger of the user) are obtained by the gesture detector(14). The first data input (data input A) corresponding to the gesture(for example, upward movement of a finger) can be generated based on theelectrode signals.

[5] The eye motion detector (15) includes a plurality of eye motiondetection electrodes (151 a, 151 b, 152 a, 152 b) on the nose pads(150R, 150L). The second data input (data input B) is generated based ona relationship between a detection signal waveform (for example, pulsesof Ch1 and Ch2 in FIG. 11) of the eye motion of the user detected by theeye motion detection electrodes and the eye motion of the user (forexample, a blink).

[6] The eye motion detection electrodes include upper and lowerelectrodes (151 a and 151 b, and 152 a and 152 b of FIG. 6) at both theright and left sides. Eye motion or eye movements in up-and-downdirections (FIGS. 8 and 9), blinks (FIG. 11), closed eyes (FIG. 12), andwinks (FIGS. 13 and 14) are detected based on a change in the detectionsignal waveform (ADC outputs from Ch1 and/or Ch2) from at least a pairof the upper and lower electrodes (151 a and 151 b and/or 152 a and 152b). Furthermore, eye motion or eye movements in right-and-leftdirections (FIG. 10) are detected from a change in the detection signalwaveform (ADC outputs of Ch0) from one of the upper and lower electrodesat the right side (for example, 151 b) and one of the upper and lowerelectrodes at the left side (for example, 152 b) of the upper and lowerelectrodes at both the right and left sides. An eyesight direction ofthe user is determined based on a detection result of the eye motion oreye movements in up-and-down directions (FIGS. 8 and 9) and/or the eyemotion or eye movements in right-and-left directions

(FIG. 10). Furthermore, the second data input (for example, data input Bused for selection of a specific letter string or icon viewed by theuser) related to the eyesight direction of the user based on a detectionresult of the blinks (FIG. 11), closed eyes (FIG. 12), and winks (FIGS.13 and 14).

[7] The eye motion detection electrodes (151 a, 151 b, 152 a, 152 b)perform sufficiently if they are simply provided with the nose pads(150R, 150L). There is no necessity of providing an additional electrodewith the bridge 103 of the glasses to contact the brow of the user. Theeye motion detection electrodes are provided with the nose pads alonewhich are adopted in ordinary glasses. Thus, a person who wears glassesdo not feel uncomfortable.

[8] The nose pads (150R, 150L) are formed of an insulating material(dielectric) such as ceramics, plastics, and rubbers. The eye motiondetection electrodes (151 a, 151 b, 152 a, 152 b) are provided with thenose pads (150R, 150L) to be separated from each other (metal fragmentsattachment, metal evaporation, conductor printing, metal ringattachment, and the like as exemplified in FIG. 5).

[9] The display (12) includes a display device (12R, 12L) to be fit inthe eye frames (101, 102). The display device is prepared by cutting atransparent plate or a lens suitable for a user to fit the shape of theeye frame and attaching film liquid crystal thereto.

Using the first data input (data input A) corresponding to the gesture,scroll and pointing can be performed with respect to the data itemsdisplayed on the display device.

[10] In addition to the above [9], using the second data input (datainput B), selection and determination can be performed with respect tothe data items displayed on the display device.

[11] A method according to one embodiment (FIG. 15) uses aneyeglasses-type wearable device with right and left eye framescorresponding to positions of right and left eyes and nose padscorresponding to a position of a nose. The eyeglasses-type wearabledevice includes a display provided with at least one of the right andleft eye frames, a gesture detector which detects a gesture indicativeof a movement of a user, and an eye motion detector which detects eyemotion or eye movement of the user.

In this method, a first data input (data input A) corresponding to thegesture detected by the gesture detector is generated (ST14). A seconddata input (data input B) corresponding to the eye motion or eyemovement detected by the eye motion detector (ST24) is generated. Aspecific process is performed based on a combination of the first datainput (data input A) and the second data input (data input B) (ST16,ST26).

[12] A method according to another embodiment uses an eyeglasses-typewearable device with right and left eye frames corresponding topositions of right and left eyes and nose pads corresponding to aposition of a nose. The eyeglasses-type wearable device includes adisplay provided with at least one of the right and left eye frames, adetector configured to detect a movement of a user (using capacitancesensors as a touch pad), and an eye motion detector which detects eyemotion or eye movement of the user.

In this method, a first data input (data input A) corresponding to thegesture detected by the detector is generated. A second data input (datainput B) corresponding to the eye motion or eye movement detected by theeye motion detector is generated. A specific process is performed basedon a combination of the first data input (data input A) and the seconddata input (data input B).

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions.

For example, the embodiments are described above to be used in theeyeglasses-type wearable device having a frame shape of ordinaryglasses. However, the embodiments can be applied to devices having ashape and structure other than such a frame shape of ordinary glasses.Specifically, a gesture detector and an eye motion detector can beprovided with eyeglasses-type wearable devices such as goggles used inskiing and snowboarding for blocking harmful ultraviolet and securingvisibility in rough conditions. Or, goggles may be used to cover theeyeglasses-type wearable device of the embodiments as shown in FIG. 3.

Furthermore, the scope of the inventions includes providing a member oran electrode (whether or not it contacts the brow of a user isirrelevant) with any optional part of the glasses such as a bridge aslong as the structures recited in the claims are maintained.

The embodiments and their variations are encompassed by the scope andoutline of the invention and by the inventions recited in claims andtheir equality. Note that a part or the whole of an embodiment of thedisclosed embodiments combined to a part or the whole of anotherembodiment of the disclosed embodiments will be encompassed by the scopeand outline of the invention.

What is claimed is:
 1. A wearable device comprising: a displayconfigured to display a three-dimensional augmented reality imageoverlapping a real world view wherein the augmented reality imageincludes a right image and a left image, and the right image and theleft image are shifted in opposite directions by adjusting an angle ofconvergence; a first detector configured to detect a direction of asightline of a user wearing the wearable device; a second detectorconfigured to detect a movement of a hand or finger of the user; and aprocessor configure to perform a process corresponding to a part of theaugmented reality image visible to the user wherein the part isdetermined based on the direction of the sightline of the user, whereinthe display is configured to change the three-dimensional augmentedreality image based on the movement of the hand or the finger of theuser.
 2. The wearable device of claim 1, wherein the display isconfigured to perform at least one of scrolling the three-dimensionalaugmented reality image and pointing the three-dimensional augmentedreality image based on the movement of the hand or the finger of theuser.
 3. A method for a wearable device comprising a display configuredto display a three-dimensional augmented reality image overlapping areal world view, wherein the augmented reality image includes a rightimage and a left image, and the right image and the left image areshifted in opposite directions by adjusting an angle of convergence, themethod comprising: detecting a direction of a sightline of a userwearing the wearable device; detecting a movement of a hand or finger ofthe user; performing a process corresponding to a part of the augmentedreality image visible to the user wherein the part is determined basedon the direction of the sightline of the user, and changing thethree-dimensional augmented reality image based on the movement of thehand or the finger of the user.
 4. The method of claim 3, wherein thechanging of the three-dimensional augmented reality image comprisesperforming at least one of scrolling the three-dimensional augmentedreality image and pointing the three-dimensional augmented reality imagebased on the movement of the hand or the finger of the user.